1. Field of Invention
This invention relates to a data display device that displays user input data and a data record/playback device that stores multimedia data with user input data and plays back or displays the stored user input data and the multimedia data in association with each other.
2. Description of Related Art
For recording important points of meetings, lectures, interviews, conversations using telephones or videophones, video image, and the like, a recordist commonly records them. For example, in a meeting, one of the participants of the meeting works as a recordist to record every statement from all participants or to record important items. However, since contents of a meeting or details of the story is not recollected from a note written in the meeting or the speaker name of a statement is not recorded, it is often difficult to make minutes after the meeting.
Conventionally, there have been proposed devices that use digital disks, digital still cameras, videotapes, semiconductor memory, and the like to store and playback data such as meetings, lectures, interviews, conversations using telephones, and videophones, as well as video images, images from monitoring cameras, and the like. Compared to having a reporter manually write down only the important points of the information to be recorded, the use of such data storage devices has the advantage of allowing voices and images to be recorded completely without omissions.
These devices can involve: recording digital signals sent via a computer network onto a storage medium; recording analog input signals from an video camera or a microphone directly to a storage medium; or encoding and converting such analog signals into digital signals.
However, when the method that records the whole meeting has no function to retrieve the recorded data, all contents of the recorded data should be played back to make minutes, that is, summarized contents of the meeting. Thus, it takes extremely long time to make minutes. In addition, these devices had a problem in that desired sections from the recorded voices or images could not be quickly searched.
To overcome this problem, tape recorders and videotape recorders have been proposed that allow easy searching of important sections by adding check-marks to important sections in the incoming audio signals or incoming image signals.
However, these check-marks only serve to specify the position of important sections. It is not possible to indicate which section of the audio signal or the image signal corresponds to each check-mark. Thus, all the audio or image signals that have been check-marked must be played back to determine the contents. Furthermore, it is also necessary to perform the awkward operation of pressing a button while someone is talking, thus distracting the user from the statement taking place.
For these reasons, there have been proposed devices in which audio signals or image signals that are input continuously are stored and played back so that they are associated with user-input data that has been entered via a pen or a keyboard by the user at selected points in time. By using such devices, the audio signal or the image signal can be recorded while the person making the recording performs pen or keyboard input in the same way that a reporter would take notes. By referring to the entered notes later, the portions of the audio or image signal to be played back can be easily selected and played back.
As this kind of art, for example, Japanese laid-open publication No. 7-226931 discloses a device that make a minutes creation work after the meeting effective by storing multimedia data entered by a meeting attendee in association with text data inputted by an recordist. By using this device, when one of the pieces of text data is selected after the meeting, the multimedia data associated with the selected text data can be easily retrieved. Thus, circumstances of the meeting can be reproduced from the meeting note written by the recordist.
For example, in Japanese laid-open publication number 7-182365, Japanese laid-open publication number 6-176171, Japanese laid-open publication number 6-343146, ACM CHI '94 Proceedings pgs. 58-64 ("Marquee: A Tool for Real-Time Video Logging") the subject matter of which are incorporated herein by reference, there are proposed devices wherein user-input data is associated with audio signals or image signals by using a time stamp. When the data is to be played back, a piece of user-input data displayed on a screen can be specified, and the audio signals or image signals recorded at the same time as the specified user-input data was recorded, can be played back.
In Japanese laid-open publication number 6-205151, the subject matter of which is incorporated herein by reference, there is disclosed a device that records audio signals or image signals while adding an index whenever user input has been interrupted for a fixed period of time. When playback is to be performed, user-input data displayed on a screen can be specified, and the audio signals or image signals from the index section corresponding to the specified user-input data is played back.
Further, Japanese laid-open publication number 7-110822 discloses a hypermedia device in which input data from the user and audio or image signals are associated with each other, and then are entered into the computer. Thus, this hypermedia device automatically creates a link between the user-input data and audio or image signals.
By using the devices disclosed in the aforementioned Japanese laid-open publication number 7-182365, 6-176171, 6-343146, 6-205151, 7-110822, and ACM CHI'94 Proceedings pgs. 58-64, the recordist can input data by using a pen or keyboard as if he wrote a note, and audio or image data are recorded. Thus, by referring to the inputted note (user-input data) later, a portion of the audio or image signals, which is desired to play back, can be easily selected and played back. This makes the creation of minutes efficient.
However, those conventional arts described above display only user-input data, such as pen data or a string entered from a keyboard, on the display screen. Data referring to the circumstances when the user-input data is entered is not displayed.
When minutes are created by selecting important points of a meeting from the recorded multimedia data and user-input data, and then editing them, if retrieval in the editing work can be performed easily and certainly by using only data displayed on a screen, retrieving work becomes much easier.
In the conventional arts described above, user-input data are displayed on a screen. When one of the displayed user-input data is specified, audio data or image data corresponding to the specified user-input data can be retrieved.
However, all the user-input data do not have the same priority. Also all the multimedia data that is recorded corresponding to the user-input data does not have the same priority. In addition, the priority of the multimedia data changes according to search condition during the editing.
Thus, the conventional arts described above simply display user-input data as a cue of retrieving. Therefore, it is difficult to know the priority or meaning of the user-input data from the display of the user-input data. Further, there is no cue to know the contents of the multimedia data that is stored corresponding to the user-input data.
Thus, in the retrieving of the conventional arts described above, the priority or meaning of each piece of the audio data or image data corresponding to each of the user-input data cannot be specified, unless audio data or image data corresponding to all the user-input data are played back. In other words, in the conventional art described above, a cue to retrieve audio data or image data is only the user-input data displayed on a screen. The user-input data is sometimes insufficient to retrieve a required audio or image data.
For example, when comments from a particular speaker is required to be played back, it is impossible to know which user-input data was recorded for a statement of the speaker, unless the user intentionally recorded the speaker who speaks at the point in time of user input. Thus, portions of audio data and portions of image data corresponding to user input data should be played back one after another, and their contents should be examined, in order to retrieve comments from the speaker.
Japanese Patent Laid-Open Hei.7-226931 described above disclose a method to store text data inputted by the user associated with the data identifying the speaker who speaks when the text is inputted. But the association therebetween is not displayed on the screen. Thus, the association between the user-input data and the comment of the speaker cannot be used during retrieving operation.
Japanese Patent Laid-Open Hei.7-182365 discloses a method for creating a series of image frames for each speaker, and a method for displaying a history of the time points of change of speakers and the speakers on the time axis. However, in the art disclosed by Japanese Patent Laid-Open Hei.7-182365, since these data are not displayed in association with user-input data, data that shows the association between the user-input data and the comment of speaker cannot be referred at the time of retrieving operation.
In addition, Japanese Patent Laid-Open Hei.7-182365 discloses a method for displaying a history of all time points of changes of speakers and the speakers on a screen. However, in a device that retrieves audio or image data by using user-input data as a cue, it is redundant to display a history of all time points of change of speakers. Since it makes display on a screen rather complicated, retrieving operation becomes difficult.